Real time ultrasound imaging method and system using an adapted 3D model to perform processing to generate and display higher resolution ultrasound image data

ABSTRACT

A method is provided for generating an ultrasound image of an anatomical region having a volume. First image low resolution image data is enhanced by adapting a 3D anatomical model to the image data to generate a second, greater, quantity of ultrasound image data in respect of the anatomical region. The enhanced volumetric information is then displayed. An anatomical model is thus used to complete partial image data thereby increasing the image resolution, so that a high resolution volumetric image can be displayed with a reduced image capture time.

RELATED APPLICATIONS

This application is the U.S. National Phase application under 35 U.S.C.§ 371 of International Application No. PCT/EP2018/066163, filed on Jun.19, 2018, which claims priority to and the of European Application No.17177771.7, filed Jun. 26, 2017. These applications are herebyincorporated in their entirety by reference herein.

FIELD OF THE INVENTION

This invention relates to an ultrasound imaging method and apparatus

BACKGROUND OF THE INVENTION

Ultrasound imaging is increasingly being employed in a variety ofdifferent applications. It is important that the image produced by theultrasound system is as clear and accurate as possible so as to give theuser a realistic interpretation of the subject being scanned. This isespecially the case when the subject in question is a patient undergoinga medical ultrasound scan. In this situation, the ability of a physicianto make an accurate diagnosis is dependent on the quality of the imageproduced by the ultrasound system.

Due to its excellent temporal resolution and its non-invasiveness,ultrasound plays an important role in cardiovascular imaging. Mostcommonly, Transthoracic Echo (TTE) imaging is used to determine leftventricular quantities such as volume or ejection fraction. To minimizemanual effort and user variability, this process has been automated byemploying anatomically intelligent model-based segmentation (Ecabert, O.et al.; IEEE Transactions on, 2008, 27, pp. 1189-1201).

The diagnostic value of acquiring 3D scans over time strongly depends onthe sampling rate. The latter is mainly restricted by the amount ofvolume data acquired for one scan.

There is thus a compromise between the speed of recording and thequality of an acquired image. However, in order to sufficiently captureand judge diagnostically relevant motion as well as abnormalitiesthereof, high frame rates are sometimes necessary. Therefore, certainclinical applications still entail the need for faster recordings thanwith a full 3D scan and sacrifice possibly valuable 3D volumeinformation by restricting the field of view.

There is therefore a need for a method and system which enables highresolution image data to be acquired in a reduced time period. Thus, theinvention aims to resolve the dilemma that both, high speed and largevolume acquisition, cannot be achieved simultaneously.

WO 2017/042304 discloses an ultrasound system in which ultrasound beamsof variable frequency are generated, with a higher frequency used whenimaging within a region of interest than when imaging outside the regionof interest. Thus, a wider field of view and higher penetration depth isused outside the region of interest, and within the region of interest ahigher resolution image is obtained.

SUMMARY OF THE INVENTION

The invention is defined by the claims.

In accordance with an aspect, there is provided a real time imagingmethod for generating an ultrasound image of an anatomical region havinga volume, the method comprising:

-   -   receiving image data for the anatomical region in the form of a        first quantity of ultrasound image data in respect of the        anatomical region volume;    -   accessing a 3D model which is a representation of the anatomical        region and which defines the spatial extent of anatomy parts of        the anatomical region;    -   adapting the 3D model to the image data; and    -   using the adapted 3D model to perform processing of the image        data thereby to generate a second, greater, quantity of        ultrasound image data in respect of the anatomical region; and    -   displaying volumetric information using the second quantity of        ultrasound image data.

This method makes use of an anatomical model to complete partial imagedata thereby increasing the image resolution, so that a high resolutionvolumetric image can be displayed with a reduced image capture time.This enables real time imaging so that spatio-temporal motion patternsmay be detected, which may be specific to a particular patient, diseaseor pathology. For example, this may be of particular interest forvalvular heart disease diagnosis and treatment planning.

The term “adapting” is thus used to relate to fitting the anatomicalmodel to the received image data, which may be coarse (spatially ortemporally) image data. The term “processing of the image data” is usedto refer to the conversion of the received (e.g. coarse) image data intohigher resolution data (with higher spatial or temporal resolution),using information from the adapted 3D model i.e. taking anatomicalinformation into account.

The 3D model may be a generic representation of the anatomical region,either for any subject, or tailored to a particular class of patients(e.g. by age, gender, size or medical condition).

The 3D model representation is for example a triangular mesh and theultrasound image comprises a set of voxels, and both are in 3D space.The model can thus be deformed to fit the low resolution image. This istermed a model adaptation.

The mesh model is created based on a training population. Thedestination resolution (of the second quantity of ultrasound data) canbe defined by the user. Typical clinical resolutions are of 0.3-1.0 mmvoxel edge length. By lowering the image capture resolution or droppingcertain parts of the data, a higher capture rate is achieved. By usingan anatomically intelligent volume completion (i.e. the processing ofthe image data), the data still has the desired clinical resolution atthe visualization stage afterwards.

Adapting the 3D model to the image data may comprise:

-   -   from the image data, generating modified image data, without        reference to the 3D model; and    -   adapting the 3D model to the modified image data.

To create this modified image data, certain voxels can be filled withthe recorded imaging information. Gaps between these voxels whichoriginate from the sparse way of data acquisition may be filled usingsimple (by which is meant non-anatomically intelligent) interpolation(such as nearest neighbor or trilinear interpolation).

The processing of the image data before any use of the anatomical modelwill be termed “modification”. Such processing makes no use of anyextracted anatomical information. This modification thus does notinvolve any delineation of regions of different characteristicanatomical or ultrasound properties.

The resulting modified image data may have steps and may be coarse stillwhen looking at the image information, but model adaptation (i.e.fitting) is still possible. The image data modification (which may be anon-anatomically intelligent interpolation) is only used for the purposeof the adaptation step, by which the volume data is fitted to theanatomical model.

The image data may comprise a set of 2D slice images, and the secondquantity of ultrasound image data comprises a 3D volumetric image withadditional image data between the 2D slice images.

Alternatively, the image data may comprise a 3D volumetric image of afirst resolution, and wherein the second quantity of ultrasound imagedata defines a 3D volumetric image of a greater, second resolution.

The method produces high resolution 3D image data either by providingadditional data between 2D slices or by providing additional data to thelower resolution 3D image data.

These two cases result from the way data is acquired. Data is typicallyacquired in polar coordinates and is then interpolated to Cartesiancoordinates either resulting in a (e.g. low) resolution image volume ora set of slices in this new Cartesian space. The model is then adaptedeither to this data directly or to a modified image (which results fromthe non-anatomically intelligent interpolation explained above) based onthe known data. After the model is adapted and therefore anatomicalregions/segments are defined, then the anatomically intelligent imageprocessing is performed to complete the image data.

The low resolution image data is for example obtained on the basis thatparts of the scan lines of a complete full volume are omitted (e.g. in achecker-board fashion). The physical beam width may be the same as wouldbe used for a full resolution image. Thus, the low resolution image forexample may have the same beam width and same focus as for a highresolution image.

There may for example be 3 to 10 image slices in each of two orthogonaldirections if the low resolution data is a subset of slices. The lowresolution image may for example comprise 25% or 50% of the scan linesif the low resolution data is based on partial scan lines.

The ultrasound system may be used in a low resolution mode when it hasthe capability of higher resolution, for the purposes of speeding upimage acquisition. Alternatively, a lower cost ultrasound system may beused at its full resolution so that the system cost is kept down for asystem able to deliver high resolution images.

The adapting the 3D model to the image data may comprise identifyinganatomical boundaries between different regions, and wherein theprocessing of the image data comprises processing data of the firstquantity of ultrasound image data within the different regions.

The processing of the image data (i.e. the data volume completion) maythus be performed within anatomical regions, but not across boundaries(transitions) between those regions, so that the boundaries can remainand maintain their distinct properties which are linked to theiranatomical meaning. A region may also share information relating to thebroader (anatomical) context, which can for example be use to assistimage data processing between the different regions i.e. in thetransitions.

The identification of regions by the adapting step may be termed“segmentation”. The regions could be distinct anatomical regions e.g.myocardium muscle tissue, but also conjunctions between two anatomicalregions/tissues. Regions are defined as areas having distinct propertiesin terms of ultrasound imaging.

Thus, similar anatomical regions are taken into account (e.g. for theimage data processing, in respect of the myocardium only those voxelsknown to be in the myocardium are used, in respect of a blood pool onlythose voxels known to be in the blood pool are used, in respect of aconjunction between muscle and blood pool only those voxels also lyingon such a conjunction are used, etc.). By defining anatomical regionsbased on the segmentation of the adapting step, restrictions are imposedon the data processing such that there is anatomically intelligentvolume completion.

The data processing thus has a spatial constraint (only usinginformation in the neighborhood of the voxel of interest) and ananatomical constraint (using information for voxels having an anatomicalmeaning similar to the voxel of interest).

The processing of the image data within the different regions maycomprise:

-   -   nearest neighbor interpolation;    -   linear interpolation; or    -   non-linear interpolation.

These different interpolation methods may be used to create theadditional image data, but within identified regions.

The processing of the image data within the regions may insteadcomprise:

-   -   interpolation based on ultrasound signal statistics in the        spatial or anatomical neighborhood.

A statistical analysis provides an approach which models the ultrasoundsignal magnitude of a voxel with the most likely value from aprobability distribution having certain learned parameters. Finally, theset of probability distributions enables an interpolation model that isused together with the model-based segmentation output to fill themissing gaps to create the higher resolution image.

The processing of the image data may comprise determining the locationand characteristics of point scatterers and a convolution with a pointspread function.

This approach for example involves randomly defining a set of pointscatterers across the ultrasound volume then convolving the resultingscatter map with a 3D spatially varying point spread function (PSF). ThePSF appearance and variance is defined by the ultrasound imagingparameters as well as the transducer.

The invention also provides a computer program comprising computerprogram code means which is adapted, when said computer program is runon a computer, to implement the method as defined above.

The invention also provides a processor arrangement for controlling thegeneration of a real time ultrasound image of an anatomical regionhaving a volume, wherein the processor arrangement is adapted to:

-   -   receive image data for the anatomical region in the form of a        first quantity of ultrasound image data in respect of the        anatomical region volume;    -   access a 3D model which is a representation of the anatomical        region and which defines the spatial extent of anatomy parts of        the anatomical region; and    -   adapt the 3D model to the image data;    -   use the adapted 3D model to perform processing of the image data        thereby to generate a second, greater, quantity of ultrasound        image data in respect of the anatomical region; and    -   display volumetric information using the second quantity of        ultrasound image data.

This processor implements the method described above.

The processor may be adapted to adapt the 3D model to the image data by:

-   -   from the image data, generating modified image data, without        reference to the 3D model; and    -   fitting the 3D model to the modified image data.

As explained above, the processor may be adapted to adapt the 3D modelto the image data by identifying anatomical boundaries between differentregions and perform processing of the image data of the first quantityof ultrasound image data within the different regions. Processing of theimage data within the different regions may then be made using: nearestneighbor interpolation; linear interpolation; non-linear interpolation;interpolation based on ultrasound signal statistics in the spatial oranatomical neighborhood; or determination of the location andcharacteristics of point scatterers and a convolution with a pointspread function.

The invention also provides an ultrasound system for generating a realtime ultrasound image of an anatomical region having a volume,comprising:

-   -   an ultrasonic transducer array, wherein the ultrasonic        transducer array is capable of emitting and receiving ultrasonic        signals which provide a first quantity of ultrasound image data        in respect of the anatomical region volume;    -   a database which stores a 3D model which is a representation of        the anatomical region and which defines the spatial extent of        anatomy parts of the anatomical region;    -   a processor as defined above; and    -   a display for displaying volumetric information using the second        quantity of ultrasound image data.

A user interface may be provided which allows a user to set an imagesampling speed.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are described in more detail and by way ofnon-limiting examples with reference to the accompanying drawings,wherein:

FIG. 1 shows an ultrasound diagnostic imaging system to explain thegeneral operation;

FIG. 2 shows 2D image slice locations through the heart;

FIG. 3 shows one 2D image of the slices of FIG. 2 ; and

FIG. 4 shows an ultrasound imaging method.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The invention will be described with reference to the Figures.

It should be understood that the detailed description and specificexamples, while indicating exemplary embodiments of the apparatus,systems and methods, are intended for purposes of illustration only andare not intended to limit the scope of the invention. These and otherfeatures, aspects, and advantages of the apparatus, systems and methodsof the present invention will become better understood from thefollowing description, appended claims, and accompanying drawings. Itshould be understood that the Figures are merely schematic and are notdrawn to scale. It should also be understood that the same referencenumerals are used throughout the Figures to indicate the same or similarparts.

Provided is a method for generating an ultrasound image of an anatomicalregion having a volume. First image low resolution image data isenhanced by adapting a 3D anatomical model to the image data to generatea second, greater, quantity of ultrasound image data in respect of theanatomical region. The enhanced volumetric information is thendisplayed. An anatomical model is thus used to complete partial imagedata thereby increasing the image resolution, so that a high resolutionvolumetric image can be displayed with a reduced image capture time.

This invention aims at improving the compromise between imageacquisition speed and image quality. By sparsely sampling only selectedview planes or scan lines instead of recording the entire volume, theacquisition time may be reduced. Anatomically intelligent model-basedsegmentation can then delineate relevant anatomical structures byfitting a 3D model to the data. Together with the echo information fromthe sampled subset, this 3D model of the region (e.g. organ) of interestcan be used to approximate and finally render a full 3D ultrasoundvolume for each point in time.

Concretely, the 3D model contributes as a boundary condition for theanatomical context and enriches a purely intensity-driven datacompletion e.g. by providing information about the expected type oftissue at a certain location and possibly its properties. Thisinformation can be used in different ways and yields more realistic, butalso case-specific results.

Overall, anatomically intelligent 3D volume completion provides a way toestimate a full 3D dataset with higher acquisition speed than for a full3D scan. It constitutes an option of 3D visualization, while stillproviding enough temporal resolution to identify anatomical effectsrelevant for diagnosis.

The general operation of an exemplary ultrasound diagnostic imagingsystem will first be described, with reference to FIG. 1 .

The system comprises an array transducer probe 10 which has a CMUTtransducer array 100 for transmitting ultrasound waves and receivingecho information. The transducer array 100 may alternatively comprisepiezoelectric transducers formed of materials such as PZT or PVDF. Thetransducer array 100 is a two-dimensional array of transducers 110capable of scanning in a 2D plane or in three dimensions for 3D imaging.In another example, the transducer array 100 may be a 1D array.

The transducer array 100 is coupled to a microbeamformer 12 in the probewhich controls reception of signals by the CMUT array cells orpiezoelectric elements. Microbeamformers are capable of at least partialbeamforming of the signals received by sub-arrays (or “groups” or“patches”) of transducers as described in U.S. Pat. No. 5,997,479(Savord et al.), U.S. Pat. No. 6,013,032 (Savord), and U.S. Pat. No.6,623,432 (Powers et al.).

Note that the microbeamformer is entirely optional. The examples belowassume no analog beamforming.

The microbeamformer 12 is coupled by the probe cable to atransmit/receive (T/R) switch 16 which switches between transmission andreception and protects the main beamformer 20 from high energy transmitsignals when a microbeamformer is not used and the transducer array 100is operated directly by the main system beamformer. The transmission ofultrasound beams from the transducer array 100 is directed by atransducer controller 18 coupled to the microbeamformer by the T/Rswitch 16 and a main transmission beamformer (not shown), which receivesinput from the user's operation of the user interface or control panel38.

One of the functions controlled by the transducer controller 18 is thedirection in which beams are steered and focused. Beams may be steeredstraight ahead from (orthogonal to) the transducer array 100, or atdifferent angles for a wider field of view. The transducer controller 18can be coupled to control a DC bias control 45 for the CMUT array. TheDC bias control 45 sets DC bias voltage(s) that are applied to the CMUTcells.

In the reception channel, partially beamformed signals are produced bythe microbeamformer 12 and are coupled to a main receive beamformer 20where the partially beamformed signals from individual patches oftransducers are combined into a fully beamformed signal. For example,the main beamformer 20 may have 128 channels, each of which receives apartially beamformed signal from a patch of dozens or hundreds of CMUTtransducer cells or piezoelectric elements. In this way the signalsreceived by thousands of transducers of a transducer array 100 cancontribute efficiently to a single beamformed signal.

The beamformed reception signals are coupled to a signal processor 22.The signal processor 22 can process the received echo signals in variousways, such as band-pass filtering, decimation, I and Q componentseparation, and harmonic signal separation which acts to separate linearand nonlinear signals so as to enable the identification of nonlinear(higher harmonics of the fundamental frequency) echo signals returnedfrom tissue and microbubbles. The signal processor may also performadditional signal enhancement such as speckle reduction, signalcompounding, and noise elimination. The band-pass filter in the signalprocessor can be a tracking filter, with its pass band sliding from ahigher frequency band to a lower frequency band as echo signals arereceived from increasing depths, thereby rejecting the noise at higherfrequencies from greater depths where these frequencies are devoid ofanatomical information.

The beamformers for transmission and for reception are implemented indifferent hardware and can have different functions. Of course, thereceiver beamformer is designed to take into account the characteristicsof the transmission beamformer. In FIG. 1 only the receiver beamformers12, 20 are shown, for simplicity. In the complete system, there willalso be a transmission chain with a transmission micro beamformer, and amain transmission beamformer.

The function of the micro beamformer 12 is to provide an initialcombination of signals in order to decrease the number of analog signalpaths. This is typically performed in the analog domain.

The final beamforming is done in the main beamformer 20 and is typicallyafter digitization.

The transmission and reception channels use the same transducer array100 which has a fixed frequency band. However, the bandwidth that thetransmission pulses occupy can vary depending on the transmissionbeamforming that has been used. The reception channel can capture thewhole transducer bandwidth (which is the classic approach) or by usingbandpass processing it can extract only the bandwidth that contains theuseful information (e.g. the harmonics of the main harmonic).

The processed signals are coupled to a B mode (i.e. brightness mode, or2D imaging mode) processor 26 and a Doppler processor 28. The B modeprocessor 26 employs detection of an amplitude of the receivedultrasound signal for the imaging of structures in the body such as thetissue of organs and vessels in the body. B mode images of structure ofthe body may be formed in either the harmonic image mode or thefundamental image mode or a combination of both as described in U.S.Pat. No. 6,283,919 (Roundhill et al.) and U.S. Pat. No. 6,458,083 (Jagoet al.) The Doppler processor 28 processes temporally distinct signalsfrom tissue movement and blood flow for the detection of the motion ofsubstances such as the flow of blood cells in the image field. TheDoppler processor 28 typically includes a wall filter with parameterswhich may be set to pass and/or reject echoes returned from selectedtypes of materials in the body.

The structural and motion signals produced by the B mode and Dopplerprocessors are coupled to a scan converter 32 and a multi-planarreformatter 44. The scan converter 32 arranges the echo signals in thespatial relationship from which they were received in a desired imageformat. For instance, the scan converter may arrange the echo signalinto a two dimensional (2D) sector-shaped format, or a pyramidal threedimensional (3D) image. The scan converter can overlay a B modestructural image with colors corresponding to motion at points in theimage field with their Doppler-estimated velocities to produce a colorDoppler image which depicts the motion of tissue and blood flow in theimage field. The multi-planar reformatter will convert echoes which arereceived from points in a common plane in a volumetric region of thebody into an ultrasound image of that plane, as described in U.S. Pat.No. 6,443,896 (Detmer). A volume renderer 42 converts the echo signalsof a 3D data set into a projected 3D image as viewed from a givenreference point as described in U.S. Pat. No. 6,530,885 (Entrekin etal.).

The 2D or 3D images are coupled from the scan converter 32, multi-planarreformatter 44, and volume renderer 42 to an image processor 30 forfurther enhancement, buffering and temporary storage for display on adisplay device 40. In addition to being used for imaging, the blood flowvalues produced by the Doppler processor 28 and tissue structureinformation produced by the B mode processor 26 are coupled to aquantification processor 34. The quantification processor 34 producesmeasures of different flow conditions such as the volume rate of bloodflow as well as structural measurements such as the sizes of organs andgestational age. The quantification processor 34 may receive input fromthe user control panel 38, such as the point in the anatomy of an imagewhere a measurement is to be made. Output data from the quantificationprocessor 34 is coupled to a graphics processor 36 for the reproductionof measurement graphics and values with the image on the display 40, andfor audio output from the display device 40. The graphics processor 36can also generate graphic overlays for display with the ultrasoundimages. These graphic overlays can contain standard identifyinginformation such as patient name, date and time of the image, imagingparameters, and the like. For these purposes the graphics processorreceives input from the user interface 38, such as patient name. Theuser interface 38 is also coupled to the transmit controller 18 tocontrol the generation of ultrasound signals from the transducer array100 and hence the images produced by the transducer array 100 and theultrasound system. The transmit control function of the controller 18 isonly one of the functions performed. The controller 18 also takesaccount of the mode of operation (given by the user) and thecorresponding required transmitter configuration and band-passconfiguration in the receiver analog to digital converter. Thecontroller 18 can be a state machine with fixed states.

The user interface 38 is also coupled to the multi-planar reformatter 44for selection and control of the planes of multiple multi-planarreformatted (MPR) images which may be used to perform quantifiedmeasures in the image field of the MPR images.

A processor arrangement may be adapted to perform any part of the methoddescribed below (with reference to FIG. 4 ). The processor arrangementmay, for instance, be included in one or more of the previouslydescribed processors, such as the controller 18, the quantificationprocessor 34 and the graphics processor 36. Alternatively, the processorarrangement may be an additional module.

In an embodiment, the method described below may be implemented by acomputer program code, included in a computer program product, which isrun on a computer.

Known ultrasound probes are technically capable of recording onlycertain planes of the entire volume. A matrix array probe, for instance,can sample only pre-defined vertical or horizontal planes or even onlysegments of these planes. Thus, given a minimally required samplingspeed, the amount of sampled volume information could therefore beadjusted to be able to meet this speed.

FIG. 2 illustrates this idea by showing a fan like arrangement of planarviews, instead of the entire volume. By activating only certain elementsof a matrix probe, the ultrasound acquisition can be restricted tomultiple planar views instead of the entire volume. A 3D anatomicallyintelligent model is then fitted in accordance with the invention to thedata and finally used to approximate the empty gaps.

FIG. 3 shows a four chamber view as an example plane with the surfacesof the 3D model cutting through the contours of the relevant anatomicalboundaries. The view thus enables segmentation of the image intodistinct anatomical regions.

FIG. 4 shows a method 200 for generating an ultrasound image of ananatomical region having a volume, for example as shown in FIG. 2 .

In step 210 image data is received for the anatomical region in the formof a first quantity of ultrasound image data in respect of theanatomical region volume. This first quantity of data corresponds to alow resolution or coarse image. It may comprise a set of 2D slice imagesor a 3D volumetric image. Typical clinical resolutions are of 0.3-1.0 mmvoxel edge length. By lowering the image capture resolution so thatlower resolution data is obtained, or dropping certain parts of thedata, a higher capture rate is achieved. The low resolution image is forexample based on selectively sampling a subset of the volumetric object,e.g. a set of planar views. Alternatively, the complete field of view ofthe 3D volume may be sampled but with a reduced density of scan lines,e.g. only every other available beam in each dimension. This for exampleyields a checker-board pattern. The physical beam width may in that casebe the same as would be used for a full resolution image. There may forexample be 3 to 10 image slices in each of two orthogonal directions ifthe low resolution data is a subset of slices. The low resolution imagemay for example comprise 25% or 50% of the scan lines.

In step 220 a 3D model is accessed which is a representation of theanatomical region. The 3D model may be a generic representation of theanatomical region (e.g. the heart), either for any subject, or tailoredto a particular class of patients (e.g. by age, gender, size or medicalcondition). The 3D model representation is for example a triangularmesh. The mesh model is for example created based on a trainingpopulation. The 3D model defines the spatial extent of the respectiveanatomy parts. Each triangle has associated information which is trainedin a data-driven way. This information provides identification inrespect of that triangle of how a typical, desired neighborhood wouldappear in an ultrasound image.

In step 240, the 3D model is adapted to the image data. This involvesfitting the anatomical model to the coarse image data.

The adaptation makes use of an anatomically intelligent segmentationalgorithm which adapts the model representation of the region (e.g.organ) of interest to the coarsely sampled data or an already naivelyinterpolated version of it (for example with interpolation between thesparsely sampled scanlines). The latter example involves firstgenerating modified image data as shown in optional step 230, withoutreference to the 3D model and then adapting the 3D model to the modifiedimage data. To create this modified image data, certain voxels can befilled with the recorded imaging information. Gaps between these voxelswhich originate from the sparse way of data acquisition may be filledusing simple interpolation (such as nearest neighbor or linearinterpolation). This image modification is before any use of theanatomical model.

This process of delineating relevant anatomical structures usingautomated by model-based segmentation approaches is known to thoseskilled in the art and is for example disclosed in Ecabert, O.; Peters,J.; Schramm, H.; Lorenz, C.; von Berg, J.; Walker, M.; Vembar, M.;Olszewski, M.; Subramanyan, K.; Lavi, G. & Weese, J. AutomaticModel-Based Segmentation of the Heart in CT Images Medical Imaging, IEEETransactions on, 2008, 27, pp. 1189-1201.In brief, the model-based segmentation involves localizing the region ofinterest, e.g. the heart, in the image. Localization may be achieved viaa completely automatic method, e.g. using a generalized Hough Transform(GHT). In such a technique, the center of gravity of the initialtriangle mesh model is placed into a 3D image according to an optimalposition obtained by a GHT. The initial mesh model is thus translatedand scaled so as to be positioned in the image.Alternatively or additionally, other localization techniques such as‘Hough forests’ and classification approaches may be used.Following localization, the segmentation routine is implemented in orderto adapt the model to the organ boundaries. The segmentation routine maybe a model-based segmentation routine which may be carried out inmultiple steps, leading from a very coarse to a finer adaption. In sucha routine, the initial mesh may be adapted rigidly by scaling, shiftingand rotating the whole mesh using a global similarity transformation.This may be followed by a global affine transformation to allow forsqueezing or stretching of the model data and a multi affinetransformation which adapts anatomical regions such as ventricles andatria individually. The mesh resulting from previous adaptationiterations may then be adapted in a deformable fashion, i.e. eachtriangle of the mesh is allowed to move independently.

The model adaptation is thus an iterative optimization, where alltriangles collectively strive to approach locations in the given imagedata for the ultrasound volume that come close to their storedinformation. In this way, a generic anatomical mesh is deformed andadapted to the given image.

The output is for example a surface-based description of the organ. Inessence, the adaptation involves deforming the model to fit the lowresolution (or modified low resolution) image. In this way, anatomicalboundaries between different regions are identified.

In step 250, the adapted 3D model is used to perform processing of theimage data thereby to generate a second, greater, quantity of ultrasoundimage data in respect of the anatomical region. This “processing of theimage data” provides conversion of the coarse image data into higherresolution data, using information from the adapted 3D model i.e. takinganatomical information into account. By using an anatomicallyintelligent volume completion, the data still has the desired clinicalresolution at the visualization stage afterwards. The desireddestination resolution can for example be defined by the user by meansof a user interface.

The processing of the image data is performed such that the anatomicalregions are taken into account. In particular, the ultrasound image datais processed region by region, rather than across boundaries betweenthose regions, so that the boundaries can remain and maintain theirdistinct properties which are linked to their anatomical meaning. Inthis way, by defining anatomical regions based on the segmentation ofthe adapting step 240, restrictions are imposed on the data processingsuch that there is anatomically intelligent volume completion (where“volume completion” refers to the step of increasing the resolution).

The processing of the image data within the different regions maycomprise:

-   -   nearest neighbor interpolation;    -   linear interpolation (e.g. trilinear interpolation in 3D volume        space); or    -   non-linear interpolation.

These different interpolation methods may be used to create theadditional image data, but within identified regions. Thus, theinterpolation is carried out locally using the anatomical boundaries.For example, for missing data in the left ventricular myocardium, anintensity is derived only from those neighbor values which are alsolocated in the left ventricular myocardium.

An identified region may also share information with other regions, forexample including information about the broader (anatomical) context.This could support interpolation for example by providing informationabout how far away is the next (and which) region. This for examplereduces the total number of regions needed, since it would reduce thenumber of transition regions that need to be defined. Thus, transitionsbetween regions could be made smoother by providing additionalinformation in the main anatomical regions.

Along similar lines, local statistics can be derived for theneighborhood of each of the missing volume parts. The statistics areestimated from the available data and constitute a local data generatingdistribution. From this distribution sampled data is used to fill in themissing parts in spatial proximity. Interpolation may additionallyexploit prior information encoded in the model, for example the modelcould include information about typical regional properties of thetissue.

An interpolation may be based on ultrasound signal statistics in thespatial or anatomical neighborhood. A statistical analysis provides anapproach which models the ultrasound signal magnitude of a voxel withthe most likely value from a probability distribution having certainlearned parameters (e.g. a Gaussian distribution with mean andvariance). These parameters can be estimated in a data-driven way fromexamples, stored in an offline database, or from the recorded or knownimage data of the current volume. The distribution can be such thatthere is one distribution per anatomical sub-region, which againexploits the knowledge from the segmentation.

Also, neighborhood information (optionally also restricted to similaranatomical regions) can be taken into account by introducing a spatialcovariance between voxels, for example using a Gaussian process.

Finally, the set of probability distributions enables an interpolationmodel that is used together with the model-based segmentation output tofill the missing gaps to create the higher resolution image.

Thus various conventional interpolation approaches may be used.

In another approach, the processing of the image data (to increase theresolution) may comprise determining the location and characteristics ofpoint scatterers and a convolution with a point spread function.

This approach is disclosed as a way of simulating ultrasound images inAlessandrini, M.; Craene, M. D.; Bernard, O.; Giffard-Roisin, S.;Allain, P.; Waechter-Stehle, I.; Weese, J.; Saloux, E.; Delingette, H.;Sermesant, M. & D'hooge, J. A Pipeline for the Generation of Realistic3D Synthetic Echocardiographic Sequences: Methodology and Open-AccessDatabase IEEE Transactions on Medical Imaging, 2015, 34, 1436-1451.While the locations of the point scatterers can be obtained from theboundaries of the adapted 3D model, their properties can be derived fromthe sparsely acquired, surrounding image intensities or/and frominformation about tissue type or mechanical properties which has beenattached to the model as additional prior knowledge.

This approach is based on randomly defining a set of point scatterersacross the ultrasound volume then convolving the resulting scatter mapwith a 3D spatially varying point spread function (PSF). The PSFappearance and variance is defined by the ultrasound imaging parametersas well as the transducer. Different local signals in the ultrasoundimage (e.g. myocardium versus blood pool) are distinguished by theirechogenicity.

This echogenicity can be simulated by either manipulating the density ofscatterers or by manipulating the amplitudes assigned to each scatterer.How, where and to which extent this manipulation is done in theultrasound volume depends on the anatomical labels for that location.This information is derived from the model-based segmentation and adatabase of typical templates. The low resolution image itself mayfunction as the reference, since parts of the image are known. Thelatter information then defines the typical amplitudedistribution/magnitude for a particular anatomical region.

In step 260, the volumetric information is displayed using the secondquantity of ultrasound image data, i.e. the higher resolution imagedata.

The image is displayed over time and this allows user interaction toexamine and diagnose the given case.

The overall method thus enables sparsely sampled data to give a higherresolution approximation of a volume for display to the user, as areplacement of the incompletely acquired data. The approximation ofsignals in unknown sub-volumes is supported by neighboring intensitiesof the recorded samples. However, prior knowledge about the typicalappearances of the imaged organ are used to provide a better estimate orto possibly to reduce further the amount of data needed.

The invention can be used in applications where high sampling rates arerequired to detect relevant spatio-temporal patterns of motion which arethen characteristic for a certain patient, pathology or disease. Anexample is given of valvular heart disease where dynamic aspects whichcan be acquired non-invasively and without ionizing radiation are highlyrelevant for diagnosis and treatment planning.

The optimal sampling speed can be set by the user or based on arecommendation for the given application. It defines the extent ofsparse sampling which is necessary to meet this requirement. Afterrecording the sparse data, the approximated volume is visualized andmade available to the user for interaction.

The invention is not restricted to the heart, but can be applied forother segmentation problems as well.

As discussed above, embodiments make use of a processor arrangement forperforming the data processing steps. The processor may be implementedby the signal processor 22 of the system of FIG. 1 . The processorarrangement can be implemented in numerous ways, with software and/orhardware, to perform the various functions required. A processor is oneexample of a controller which employs one or more microprocessors thatmay be programmed using software (e.g., microcode) to perform therequired functions. A controller may however be implemented with orwithout employing a processor, and also may be implemented as acombination of dedicated hardware to perform some functions and aprocessor (e.g., one or more programmed microprocessors and associatedcircuitry) to perform other functions.

Examples of controller components that may be employed in variousembodiments of the present disclosure include, but are not limited to,conventional microprocessors, application specific integrated circuits(ASICs), and field-programmable gate arrays (FPGAs).

In various implementations, a processor or controller may be associatedwith one or more storage media such as volatile and non-volatilecomputer memory such as RAM, PROM, EPROM, and EEPROM. The storage mediamay be encoded with one or more programs that, when executed on one ormore processors and/or controllers, perform at the required functions.Various storage media may be fixed within a processor or controller ormay be transportable, such that the one or more programs stored thereoncan be loaded into a processor or controller.

Other variations to the disclosed embodiments can be understood andeffected by those skilled in the art in practicing the claimedinvention, from a study of the drawings, the disclosure, and theappended claims. In the claims, the word “comprising” does not excludeother elements or steps, and the indefinite article “a” or “an” does notexclude a plurality. The mere fact that certain measures are recited inmutually different dependent claims does not indicate that a combinationof these measures cannot be used to advantage. Any reference signs inthe claims should not be construed as limiting the scope.

The invention claimed is:
 1. A real time imaging method for generatingan ultrasound image of a plurality of different anatomical regionswithin a volume, the method comprising: receiving low resolution imagedata for the anatomical regions in the form of a first quantity ofultrasound image data in respect of the anatomical regions within thevolume; accessing a 3D model which is a representation of the anatomicalregions and which defines the spatial extent of anatomy of theanatomical regions by boundaries; adapting the 3D model to the lowresolution image data; and using the adapted 3D model to performprocessing of the image data within each region and avoid processingtogether image data of different regions separated by a boundary therebyto generate a second, greater, quantity of ultrasound image data ofhigher resolution in respect of the anatomical regions; and displayingvolumetric information using the second quantity of ultrasound imagedata at a resolution higher than that of the low resolution.
 2. A methodas claimed in claim 1, wherein adapting the 3D model to the image datacomprises: from the image data, generating modified image data, withoutreference to the 3D model; and adapting the 3D model to the modifiedimage data.
 3. A method as claimed in claim 1, wherein: the image datacomprises a set of 2D slice images, and the second quantity ofultrasound image data comprises a 3D volumetric image with additionalimage data between the 2D slice images; or the image data comprises a 3Dvolumetric image of a first resolution, and wherein the second quantityof ultrasound image data defines a 3D volumetric image of a greater,second resolution.
 4. A method as claimed in claim 1, wherein theadapting the 3D model to the image data comprises identifying anatomicalboundaries between the different regions, and wherein the processing ofthe image data comprises processing data of the first quantity ofultrasound image data within the different regions.
 5. A method asclaimed in claim 4, wherein the processing of the image data within thedifferent regions comprises: nearest neighbor interpolation; linearinterpolation; or non-linear interpolation.
 6. A method as claimed inclaim 4, wherein the processing of the image data within the regionscomprises: interpolation based on ultrasound signal statistics in thespatial and anatomical neighborhood.
 7. A method as claimed in claim 1,wherein the processing of the image data comprises determining thelocation and characteristics of point scatterers and a convolution witha point spread function.
 8. A tangible, non-transitory computer readablemedium comprising computer executable instructions which, when saidcomputer executable instructions are run on a computer, cause thecomputer to implement the method of: receiving low resolution image datafor the anatomical regions in the form of a first quantity of ultrasoundimage data in respect of the anatomical regions within the volume;accessing a 3D model which is a representation of the anatomical regionsand which defines the spatial extent of anatomy of the anatomicalregions by boundaries; adapting the 3D model to the low resolution imagedata; and using the adapted 3D model to perform processing of the imagedata within each region and avoid processing together image data ofdifferent regions separated by a boundary thereby to generate a second,greater, quantity of ultrasound image data of higher resolution inrespect of the anatomical regions; and displaying volumetric informationusing the second quantity of ultrasound image data at a resolutionhigher than that of the low resolution.